Field of the Invention
A remote control system enables a first (or xe2x80x9clocalxe2x80x9d) computing device to be controlled via a second (or xe2x80x9cremotexe2x80x9d) computing device.
U.S. Pat. No. 5,949,412 issued to Huntsman, entitled xe2x80x9cComputer Remote Control Systemxe2x80x9d, proposed a remote control system where the client is a Web browser. A screen image of the controlled application is transferred as a bitmap image displayable within a Web browser (GIF file).
His solution lacks the following:
client graphical user interface (GUI) is a bitmap image and not made of logical elements;
no on-the-fly conversion;
no continuous update of client screen;
no customization and adaptation of the client widgets;
his invention used a click-able image to send coordinates to the server, however, he does not explain how the image is constructed nor does he claim that the image is constructed automatically;
while mouse input is supported, keyboard input is not, though keyboard capabilities are mentioned in passing;
the client uses HTML, not DHTML, consequently, there are no event handlers (only coordinate click events);
no collaboration;
no support for 3-tier architecture (our FIG. 1).
Other prior art remote control systems had similar limitations and further limitations:
First, they generally involve a bit-to-bit representation of the first computing device""s GUI on the second computing device. This is an inefficient method which does not lend itself to customization: the GUI on the second computing device has to be the same as that on the first computing device.
Second, the communications are binary, which is heavier than logical, text based communications. For example, if the first GUI changes to show new text, the image of that text must be sent to the second computing device. Whereas if the communication were logical, simply the text would be passed.
Third, since the second GUI is just a graphic, all its logic must come from the first computing device. For example, if the remote user clicks on a button, that event must be sent to the first computing device which then sends back an image of the button being recessed. Whereas if the second computing device had display level logic, it would know to display the clicking without having to contact the first computing device.
Fourth, binary communications are not xe2x80x9cfirewall friendlyxe2x80x9d. Generally a firewall must be reconfigured to allow a remote control system to operate across it.
Fifth, since the binary communications format of prior remote control applications is less secure, when a first computing device is on a LAN, it must be on the gateway to the LAN. In other words, the first computing device cannot be a workstation on the LAN because that might compromise security.
Sixth, until now it has not been possible to separate the GUI logic of desktop applications from the program logic as it is with client-server applications.
Seventh, because the previous remote control systems used xe2x80x9cbit-to-bitxe2x80x9d communications, there were sometimes problems getting the system to work correctly when the second computing device was a different software and/or hardware platform than the first computing device.
The present invention differs substantially from prior solutions in that the user interface (UI) of the first computing device is analysed and a second, logically equivalent, UI is generated and sent to the second computing device. A user can thereby operate the first computing device via the second UI on the second computing device. In a preferred embodiments of the present invention the UI is specifically a GUI (graphical user interface). The logically equivalent GUI is constructed, in these embodiments, of DHTML which is displayed in a browser on the second (remote) computing device.
To illustrate how the logical GUI used in the present invention differs from the image-based systems used in prior remote control systems, consider the example of Microsoft Corporation""s Windows (R). There the GUI consists of objects, such as buttons. When the user clicks a button, the button display changes to show itself appearing to be recessed and then return to its original state. At the same time the button might send a xe2x80x9cclicked-onxe2x80x9d message to the underlying program. The underlying program might then reply by changing the text or color of the button.
In some operating systems the GUI""s structure can be detected and analyzed (e.g., via the Windows HWND handle). Based on this analysis, the second, logically equivalent UI mentioned above can be built. The second UI becomes the new interface to the first computing devicexe2x80x94 In effect, the present invention converts a desktop application into a two tier application (client-server).
The second UI is as platform-independent as possible so as to enable it to be displayed on a wide range of second computing devices. For example, the second GUI can be built with a dynamic markup language such as DHTML, which can be viewed in most Web browsers, irrespective of the platform the browser is running on.
Therefore, one of the major objects of the present invention is to analyze a program""s UI and build a functional universal copy by which a remote computing device can control the original program.
Another object of the present invention is to effect this process on-the-fly, i.e., when a user of the second computing device opens a new program on the local computing device for remote control, the present invention automatically analyzes the first GUI, generates the second GUI, and sends it to the second computing device. No user intervention is required.
To summarize some of the major innovations of the present invention: it detects (on the fly) the appearance and logic of the UI of a local program (application or operation system) on a first computing device, transfers control of the local program to the logical UI running on the second computing device, and enables post-processing customization of the logical UI, its interaction with the user, and its interaction with the local program. And these operations can be performed on local programs that were not designed for client-server operation.
Some of the major advantages of the present invention are as follows:
First, a first GUI is converted to a second, logically equivalent GUIxe2x80x94 not a bit-to-bit image. This conversion is done on-the-fly: as a layout (window) is displayed in the first GUI, a logically equivalent duplicate GUI is generated and transferred to the second computing device where it is displayed.
Second, less data needs to be transferred since only logic is transferred, not pixel changes. For example, if the local program changes the text on a GUI object, the remote client is sent just an instruction to change the test displayed rather than a completely new image of the object including the new text.
Third, the remote client can display results of user input itself. For example, when a user clicks a button, the DHTML page handles displaying the visual effect without having to be sent an image of the button going in and then another image of the button coming back out.
Fourth, since the remote GUI is separate from the local program, the second GUI can be customized (after the on-the-fly creation) in various respects, including: the appearance/location of GUI objects, the handling of user actions, and/or the handling of events (messages) from the local application.
Fifth, the DHTML is sent to the remote client as test via the HTTP protocol. This is a well known, secure protocol that conforms to security requirements. For example, firewalls normally enable HTTP data communications. Therefore in LAN installations, the first computing device is not restricted for security reasons to a gateway, and instead can be any workstation on the LAN.
Sixth, the remote GUI will display correctly in any second computing device capable of displaying a DHTML web page, regardless of whether that second computing device is a different type of hardware, (e.g., mobile phone, personal digital assistant, etc.) and/or has a different resolution than the first computing device.
Seventh, when a second computing device joins an active remote control session it receives the second GUI in its current state.
Eighth, remote GUIs for multiple local programs can be combined in a single remote GUI.
Ninth, multiple instances of a remote GUI for a given local program can run simultaneously on multiple remote computing devices, enabling active or passive collaboration.
Tenth, the second GUI can be frozen at any timexe2x80x94when unfrozen it will update to display the current state of the local program.
There is thus provided in accordance with a preferred embodiment of a method for remotely controlling a first computing device from at least one of a plurality of second computing devices, the first computing device having a user interface and a data communications connection to to the second computing device, the second computing device adapted to present a user interface, the method comprising:
analyzing the static and dynamic logic of the first computing device""s user interface;
creating an equivalent user interface in a platform-independent format, wherein said static and dynamic logic is substantially replicated;
sending said equivalent user interface to the second computing device where it is implemented;
handling user input to said equivalent user interface, including sending output to the first computing device;
handling output originating in the first computing device, said handling including sending output to said equivalent user interface on the second computing device;
thereby enabling control of the first computing device from the second computing device.
Furthermore, in accordance with another preferred embodiment there is provided:
A method for remotely controlling a first computing device from at least one of a plurality of second computing devices, the first computing device having a graphical user interface and a data communications connection via a Web server to the second computing device, the second computing device adapted to display a Web browser, the method comprising:
analyzing the static and dynamic logic of the first computing device""s graphical user interface and creating an equivalent graphical user interface in a Web browser supported format, wherein said static and dynamic logic is substantially replicated;
sending said equivalent graphical user interface to the second computing device where it is displayed in the Web browser;
detecting changes in the status of the first computing device and sending those changes to said equivalent graphical user interface;
receiving control input from said equivalent graphical user interface and executing it on the first computing device;
thereby enabling control of the first computing device from the second computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said equivalent graphical user interface is created in real time.
Furthermore, in accordance with another preferred embodiment of the present invention, said equivalent graphical user interface is created automatically.
Furthermore, in accordance with another preferred embodiment of the present invention, said equivalent graphical user interface""s visual characteristics, its input handlers, and its output handlers is customizable.
Furthermore, in accordance with another preferred embodiment of the present invention, the most recent state of the first computing device is continuously sent to said equivalent graphical user interface in response to polling from said equivalent graphical user interface.
Furthermore, in accordance with another preferred embodiment of the present invention, the most recent state of the first computing device is sent to any additional computing device that joins the remote control session.
Furthermore, in accordance with another preferred embodiment of the present invention, the state of said equivalent graphical user interface can be frozen at any time and when unfrozen is immediately update to the current state of the first computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, a software component on the first computing device performs said analysis of the first computing device""s graphical user interface.
Furthermore, in accordance with another preferred embodiment of the present invention, a software component on the first computing device monitors output events from the first computing device and based on said dynamic and static analysis, determines what action to take.
Furthermore, in accordance with another preferred embodiment of the present invention, a software component on the first computing device monitors input events from the second computing device and handles those events based on said dynamic and static analysis.
Furthermore, in accordance with another preferred embodiment of the present invention, a software component on said Web server mediates communication between the first computing device and the said equivalent graphical user interface.
Furthermore, in accordance with another preferred embodiment of the present invention, said Web server is on the first computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said Web server is on a separate hardware device and has data communication connections with the first computing device and with the second computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said data communications connection between said Web server and the first computing device is a LAN.
Furthermore, in accordance with another preferred embodiment of the present invention, there if provided:
A system for remotely controlling a first computing device from at least one of a plurality of second computing devices, the first computing device having a graphical user interface and a data communications connection via a Web server to the second computing device, the second computing device adapted to display a Web browser, the system comprising:
a window analysis system that analyzes the static and dynamic logic of the first computing device""s graphical user interface and creates an equivalent graphical user interface in a Web browser supported format, wherein said static and dynamic logic is substantially replicated;
a mediation system that:
sends said equivalent graphical user interface to the second computing device where it is displayed in the Web browser, and sends updated information about the state of the first computing device to the equivalent graphical user interface, and receives control input from the equivalent graphical user interface;
a visual status monitoring system that detects changes in the status of the first computing device and sends those changes to said mediation system for transfer to the second computing device;
a command executing system that receives said control input from said mediation system and executes it on the first computing device;
thereby enabling control of the first computing device from the second computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said equivalent graphical user interface is created in real time.
Furthermore, in accordance with another preferred embodiment of the present invention, said equivalent graphical user interface""s visual characteristics, its input handlers, and its output handlers can be customized.
Furthermore, in accordance with another preferred embodiment of the present invention, the most recent state of the first computing device is continuously sent to said equivalent graphical user interface in response to polling from said equivalent graphical user interface.
Furthermore, in accordance with another preferred embodiment of the present invention, the state of said equivalent graphical user interface can be frozen at any time and when unfrozen is immediately updated to the current state of the first computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said mediation system sends an updated equivalent graphical user interface to any additional computing device that joins the remote control session.
Furthermore, in accordance with another preferred embodiment of the present invention, said Web server is on the first computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said Web server is on a separate hardware device and has data communication connections with the first computing device and with the second computing device.
Furthermore, in accordance with another preferred embodiment of the present invention, said data communications connection between said Web server and the first computing device is a LAN.